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Harvard Family Research Project’s series of Out-of-School Time 
Evaluation Snapshots distills the wealth of information compiled 
in our Out-of-School Time Program Evaluation Database 1 into a 
single report. Each Snapshot examines a specific aspect of out-of- 
school time (OST) evaluation. This Snapshot describes the com- 
mon data collection methods used by current out-of-school time 
programs to evaluate their implementation and outcomes. 

Once the domain of researchers, evaluation has become 
an integral part of daily life — and a key accountability re- 
quirement — for out-of-school time (OST) programs and pro- 
fessionals. Many programs use a variety of methods to collect 
information on community needs, program operations, and 
youth outcomes. This diversity of data collection options is 
one important reason for all OST professionals to understand 
the what, why, and how of data collection methods. This 
knowledge will help them make informed decisions about the 
best strategies for collecting data appropriate for their unique 
program characteristics and for the evaluation questions they 
wish to address. 

This Snapshot describes common data collection meth- 
ods and offers guidelines for their use. It provides detailed 
information about using surveys and questionnaires, 
interviews and focus groups, observations, tests and 
assessments, and secondary sources and data reviews. 
In describing each method, this Snapshot provides real-world 
examples from program evaluations represented in Harvard 
Family Research Project’s Out-of-School Time Program 
Evaluation Database. 

EVALUATION TERMINOLOGY 
USED IN THIS SNAPSHOT 

When it comes to the language of evaluating programs, 
stakeholders use a variety of terms. The following definitions 
provide a common language for this Snapshot. These terms 
are important for understanding how data collection methods 
fit into the context of program evaluation. 
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Harvard Family Research Project’s 
Out-of-School Time Program 
Evaluation Database 

The Harvard Family Research Project (HFRP) Out-of- 
School Time Program Evaluation Database contains 
profiles of out-of-school time (OST) program evalua- 
tions. Its purpose is to provide accessible information 
about previous and current evaluations to support the 
development of high quality evaluations and programs in 
the OST field. 

Types of Programs Included in the Database 

Evaluations in the database meet the following three 
criteria: 

1. The evaluated program or initiative operates during 
out-of-school time. 

2. The evaluations aim to answer a specific evaluation 
question or set of questions about a specific program 
or initiative. 

3. The evaluated program or initiative serves children 
between the ages of 5 and 19. 

Types of Information Included in the Database 

Each profile contains detailed information about the 
evaluations as well as an overview of the OST program 
or initiative itself. Web links to actual evaluation reports, 
where available, are also provided, as are program and 
evaluation contacts. 

How to Use the Database 

The database is located in the OST section of the 
HFRP website at www.gse.harvard.edu/hfrp/projects/ 
afterschool/evaldatabase.html. The search mechanism 
allows users to refine their scan of the profiles to spe- 
cific program and evaluation characteristics and findings 
information. 

The Scan for This Snapshot 

For this review, we conducted a manual scan of all 
evaluation profiles and recorded all the data collection 
measures used by each program. 
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Evaluation - an overall assessment of whether a program 
achieves its goals, how it works, and what adjustments may 
improve its results 

Outcome - a bottom-line condition of well-being for 
children, families, or communities. It is a broadly defined, 
fundamental condition that government and citizens consider 
essential. An outcome may also be referred to as a result. 

Sample - the group of individuals or other units (e.g., pro- 
grams, communities, etc.) studied. A sample is often a subset 
of the total population served by a program. 

Data collection method - the strategy and system used to 
gather information on participants, programs, and other ele- 
ments of the evaluation 

Data source - the individual or institution from which the 
evaluation data are collected (e.g., participants, parents, 
school records, etc.) 

Quantitative data - numeric information that is subject to 
statistical analysis 

Qualitative data - text-based information that provides 
descriptive details, often collected from interviews, focus 
groups, or observations 

DATA COLLECTION METHODS 

The following six data collection methods are used by OST 
programs across the country to collect both formative and 
summative evaluation information. The following information 
is provided below about each data collection method: its 
primary purpose, the type of information it can collect (based 
on our review of the evaluations posted in our OST Program 
Evaluation Database), its advantages and challenges, and any 
relevant additional information. 

Method I: Surveys and Questionnaires 
Purpose: Surveys and questionnaires are collected by evalua- 
tors to gather specific information from participants, families, 
staff and administrators, teachers, community members, and 
other stakeholders. Data collected often include demographic 
information, satisfaction levels, and opinions of the program. 

Surveys and questionnaires are usually administered on pa- 
per, in a structured or semi-structured format. Respondents 
often choose from among a set of forced-choice, or provided, 
responses. These can include yes/no or scaled responses. 
Surveys and questionnaires can be administered in person, by 
mail, over the phone, or via email/Internet. 

Type of Information Collected 

■ Youth and family demographics 

■ Program characteristics (activities, staffing, level of paren- 
tal involvement) 

■ Youth’s program participation (how often, for how long, 
in which program activities) 



IN FOCUS: 

Surveys in OST Program Evaluations 

The Across Ages program in Philadelphia, Pennsylva- 
nia, uses older adults as mentors for youth. Mentors 
help at-risk youth develop awareness, self-confidence, 
and skills to help resist drugs and overcome obstacles. 
The Across Ages program evaluation surveyed all youth 
participants, as well as a comparison group of youth, be- 
fore and after the program. Youth surveys assessed per- 
ceptions of the mentoring relationships, attitudes about 
the elderly, and feelings about school and the future. 

Many OST evaluations use creative strategies to make 
their surveys accessible. California’s After School 
Achievement Program addressed an important 
barrier to surveying parents by offering the survey in 
multiple languages. The North Carolina Support 
Our Students initiative, which provides funding to 
local OST programs, used Internet-based surveys to 
collect data from program directors on program activi- 
ties, attendance, and other program-related data. 

For the full profile of these evaluations see the Harvard 
Family Research Project Out-of-School Time Program 
Evaluation Database at www.gse.harvard.edu/hfrp/projects/ 
afterschool/evaldatabase.html. 



■ Youth’s and families’ perceptions of and satisfaction with 
the program 

■ Youth outcomes (academic adjustment and achievement, 
social skills, risk-taking behavior, attitudes, self-concept, 
employability, etc.) 

Advantages 

■ Less time consuming and expensive to administer than 
other methods 

■ Can be administered to large groups of individuals 

■ Effective for assessing program satisfaction 

Challenges 

■ Data entry and analysis can be time consuming 

■ May be difficult to receive completed surveys from stake- 
holders. A range of incentives can be offered on return of 
completed surveys to boost the likelihood of response, 
from financial compensation for providers and families to 
parties for youth participants. 

Additional Information: Surveys and questionnaires are 
the most commonly used method for collecting data by pro- 
grams in the OST Program Evaluation Database. 

Method 2: Interviews and Focus Groups 
Purpose: Interviews and focus groups are most often used 
to gather detailed, qualitative descriptions of how programs 
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operate and how stakeholders perceive them. Interviews are 
conducted one-on-one, while focus groups are conducted 
in small groups. Both are usually conducted with targeted 
samples of stakeholders, such as staff, administrators, youth, 
families, funders, and community members. 

Interviews and focus groups can be conducted in person 
or by phone. Questions are generally open-ended and 
responses are documented in thorough, detailed notes or 
transcription. However, some interviews use structured 
quantitative response categories. 

Type of Information Collected 

■ Student, family, and community background (demograph- 
ics, alternative or prior OST arrangements) 

■ Program characteristics (goals, needs, implementation, 
staff hiring and training, program accessibility, community 
involvement) 



IN FOCUS: 

Interviews and Focus Groups 
in OST Evaluations 

Austin Eastside Story After-School Program 

(AES) aims to promote academic, social, and cultural 
development for youth, as well as parental empower- 
ment, for families in East and Northeast Austin, Texas. 
An AES evaluation of program implementation and 
participant outcomes included one-on-one interviews 
with approximately half of the program teachers and 
a sample of parents based on their time availability. In 
addition, focus groups were conducted with a sample 
of children in grades three and higher. 

Bayview Safe Haven, a community-based after 
school program for at-risk youth in San Francisco’s 
Bayview/Hunter’s Point neighborhood, provides struc- 
tured activities in academics, recreation, vocational 
training, and life skills to help youth stay in school and 
out of the criminal justice system. The first step in the 
impact evaluation was to interview all participating 
youth after they had attended the program for 2 to 3 
days about their current life situations and future goals. 
Evaluators also interviewed 38 program youth after 
sustained participation to assess youth perceptions of 
the program’s impact. 

Evaluators of the Core Arts Program in Mississippi 
interviewed staff and administrators to identify the 
most highly valued outcome indicators and to create a 
youth survey designed to measure these outcomes. 

For the full profile of these evaluations see the Harvard 
Family Research Project Out-of-School Time Program 
Evaluation Database at www.gse.harvard.edu/hfrp/projects/ 
afterschoollevaldatabase.html. 



■ Youth’s program participation (how often, youth’s goals in 
the program) 

■ Youth’s and families’ perceptions of and satisfaction with 
the program 

■ Youth outcomes (academic adjustment, social skills, at- 
titudes, self-concept) 

Advantages 

■ Provide rich data that paint a broad picture 

■ May highlight issues not previously considered or informa- 
tion that is useful for interpreting quantitative data col- 
lected through other methods 

■ Small focus groups may increase the comfort level of par- 
ticipants 

Challenges 

■ Can be difficult to elicit participation from individuals who 
have time constraints 

■ Interviews and large focus groups may intimidate some 
participants 

■ Documentation and analysis can be time consuming and 
may require the help of someone versed in qualitative 
analysis 

Method 3: Observations 

Purpose: Observations are a generally unobtrusive method 
for gathering information about how the program or initiative 
operates. They are usually conducted by external evaluators 
or researchers and are often used to verify and supplement 
information gathered through other methods. This is often 
a method used to assess program quality standards such as 
those in the School-Age Care Environmental Rating Scale 
(SACERS). 2 

Observations can be highly structured, with protocols for 
recording specific behaviors at specific times, or unstructured, 
taking a “look-and-see” approach. They are most reliable 
when they are conducted over a period of time to minimize 
the chances of the observation day(s) being atypical. 

Type of Information Collected 

■ Program characteristics (implementation, activities, inter- 
personal interactions, administration and management, 
health and safety) 

Advantages 

■ Provide highly detailed information from an external per- 
spective on what actually occurs in programs 

■ Trained evaluators may provide less biased descriptions 
than program staff or stakeholders 

Challenges 

■ Can be time consuming, labor intensive, and expensive 

■ Observers must be trained and be consistent with one 
another 

■ Observations conducted on a sample of days may not 
represent the range of program practices and experiences 
over time 
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IN FOCUS: 

Observations in OST Evaluations 

Some large-scale programs, including the San Fran- 
cisco Beacons Initiative (SFBI), have designed 
their own observation protocols. SFBI is a network 
of school-based centers that provide comprehensive 
services and enrichment activities for youth and other 
community members. The SFBI evaluation included 
observations of peer interactions, staff-participant 
interactions, and opportunities for youth autonomy 
and decision making. 

Other programs choose to use standardized observa- 
tion instruments. The evaluators of the Hawaii After- 
School Plus Program conducted observations 
according to the Assessing School-Age Care Quality 
(ASQ) instrument. This instrument was developed 
specifically for school-age child care programs and uses 
a 5-point scale to rate program characteristics. 

For the full profile of these evaluations see the Harvard 
Family Research Project Out-of-School Time Program 
Evaluation Database at www.gse.harvard.edu/hfrp/projects/ 
afterschoollevaldatabase.html. 



Method 4: Tests and Assessments 
Purpose: Tests and assessments are developed or used spe- 
cifically for the program evaluation to quantify characteristics 
of the program, participants, or outcomes. Examples include 
achievement tests and psychological tests (e.g., assessments 
of depressive symptoms and self-esteem). They may be stan- 
dardized or created by program evaluators for the specific 
program. 

Type of Information Collected 

■ Program implementation (staffing models, activity offer- 
ings, management and organizational strategies) 

■ Youth outcomes (academic achievement, self-esteem, 
mental health and well-being, social skills and development, 
risk-taking behavior, career maturity, fitness) 

Advantages 

■ Often more valid and reliable than perceptions or opinions 

■ Comparing scores before and after the program is a strong 
method for assessing whether outcomes actually changed 
over time 

Challenges 

■ Can be costly and time intensive 

■ May require scoring by an external source 

Additional Information: For more information on 
standardized assessments, see HFRP’s Snapshot 6, due to 
be available in fall 2004. (To be notified when it is avail- 



able sign up for our out-of-school time updates email at 
www.gse.harvard.edu/hfrp/subscribe.html.) 

Method 5: Document Reviews 
Purpose: Document reviews analyze existing program 
records and other documents not gathered or developed spe 
cifically for the evaluation. Examples include recruitment and 
attendance records, budget, staff records, and annual reports. 
They are particularly useful for documenting implementation. 

Type of Information Collected 

■ Youth, family, and community demographics 

■ Program characteristics (implementation, activities and 
curricula, budget) 

■ Youth’s program participation (how often and for how 
long) 

■ Youth characteristics before and after the program (aca- 
demic achievement, school attendance, limited English 
proficiency status) 

Advantages 

■ Records are tailored to programs 

■ Save on evaluation time and costs 

■ May elicit a high degree of accuracy from staff if they are 
also used for accountability purposes 

Challenges 

■ May not be available or applicable for some indicators 

■ May be incomplete due to staff time constraints 



IN FOCUS: 

Document Reviews in 
OST Evaluations 

BELL’s (Building Educated Leaders for Life) After- 
School Instructional Curriculum (BASICs) is a 
30-week extended-day tutorial program operated 
in Boston, New York, and Washington, D.C. This 
program aims to improve participants’ academic per- 
formance, motivation, self-concept, and social skills by 
partnering youth with tutors and mentors recruited 
from high schools, colleges, and the community. Evalu- 
ators collected data before and after the program to 
assess changes in academic outcomes. Part of this pro- 
cess involved reviewing participant portfolios, which 
were compiled throughout the program year and used 
by staff to create youth progress reports. Portfolios in- 
cluded writing samples, book reports, assignments and 
quizzes, tutors’ notes, and youth’s self-reported goals. 

For the full profile of these evaluations see the Harvard 
Family Research Project Out-of-School Time Program 
Evaluation Database at www.gse.harvard.edu/hfrplprojects/ 
afterschool/evaldatabase.html. 
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Method 6: Secondary Sources and 
Data Reviews 

Purpose: Secondary sources and data reviews use existing 
documents or data that were originally collected for purposes 
other than the program evaluation or documentation, but 
which are useful for the evaluation. Examples include achieve- 
ment data, standardized test scores, court records, and com- 
munity demographic data. 

They can also include previous similar studies, which are 
used to compare program youth with other American youth 
(e.g., National Education Longitudinal Survey). 

Type of Information Collected 

■ Youth, family, and community background (demographics, 
neighborhood income, criminal history, mental health 
history) 

■ Program characteristics (implementation, program-school 
connections) 

■ Youth academic outcomes (grades, test scores, attendance, 
disciplinary actions, grade promotion, course enrollment) 



IN FOCUS: 

Secondary Sources and Data Reviews 
in OST Evaluations 

Initiated in 2000 in Columbus, Ohio, the mission of the 
Capital Kids (formerly Cap City Kids) after school 
program is to provide a safe, caring environment where 
children can increase their academic, interpersonal, 
and social skills, and to promote connections between 
schools, families, and communities. An ongoing pro- 
gram evaluation assesses change in academic outcomes 
by using school records data. Teachers’ reports of 
reading, math, and homework completion grades are 
collected before, during, and after the program, and 
transformed into a standardized scale. These scores 
are then compared for participating youth and a com- 
parison group of nonparticipating youth to evaluate 
whether participating youth showed greater gains in 
academic achievement. 

In 1997-1998, the 4-H Youth Development Pro- 
gram — Cornell Cooperative Extension used the 
data review method to establish a comparison group 
for program youth. Using a large dataset from the 
Search Institute allowed the researchers to compare 
4-H participants with other youth from similar back- 
grounds and to evaluate whether 4-H youth had more 
positive outcomes over a 2-year period. 

For the full profile of these evaluations see the Harvard 
Family Research Project Out-of-School Time Program 
Evaluation Database at www.gse.harvard.edu/hfrp/projects/ 
afterschoollevaldatabase.html. 



■ Community outcomes (crime rates, creation of commu- 
nity partnerships) 

Advantages 

■ May be less biased than perceptions or opinions 

■ Can save administrative time and costs 

Challenges 

■ Obtaining records often requires special permission from 
parents and school officials 

■ Some national datasets charge access fees 

Additional Information: Programs in the OST Program 
Evaluation Database primarily use secondary sources to 
collect academic achievement data and, to a lesser extent, 
information on program implementation. 

DATA COLLECTION TIPS: 

ISSUES TO CONSIDER 

Using Single Versus Multiple Methods 

No one data collection method is ideal for every situation. 

For this reason, it is preferable to use multiple methods 
whenever possible. Using multiple methods to assess the 
same outcomes (e.g., using surveys and document review to 
assess program management) provides a richer, more detailed 
picture. It also illuminates inconsistencies between methods 
and reduces the chance of bias caused by a particular method. 
According to the scan of the OST Program Evaluation Data- 
base, multiple methods are more often used by large-scale 
evaluations of multisite programs, such as the After School 
Achievement Program and California’s Communities Organiz- 
ing Resources to Advance Learning initiative. This may reflect 
the fact that using multiple methods requires more resources, 
which tend to be more available in large programs. 

Selecting Data Sources 

Equally important to selecting a data collection method is 
selecting a data source, or information provider. Data can be 
collected from youth, families, staff, funders, educators, and 
other stakeholders. Certain data sources lend themselves 
more easily to certain data collection methods; for example, 
to assess parent satisfaction, surveys are popular because 
they can be mailed to hard-to-reach parents. As with data 
collection methods, it is recommended that evaluations use 
several data sources. However, programs may already have 
some of the evaluation information they need. Conducting an 
inventory of what programs already have and what is already 
available in the community can reduce data burden as well as 
data collection costs. 

Selecting a Sample of Individuals 

The choice of data collection methods is also affected by 
the sample to be studied. Some methods are well suited to 
collecting data from all participants (e.g., surveys), while oth- 
ers are better suited to a smaller group that represents the 
diversity of all participants (e.g., focus groups). The choice 
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of sample is in turn affected by the size of the program to be 
evaluated. In general, including all participants produces the 
most reliable results but may limit the type and amount of 
data collected because of the cost implications of doing so. 

Collecting Data Before and After Program 
Participation and Implementation 

Using the same data collection method to gather information 
before the start of the program and after its completion (also 
known as a pre/posttest design) provides the opportunity to 
determine whether some characteristic changed during the 
course of the program. This can suggest that the program 
played a role in effecting the change; however, unless a 
program uses random assignment as part of its pre/posttest 
design, this method cannot establish that the program caused 
the change, because other unmeasured factors may have been 
responsible. 

Cost Considerations 

Selecting data collection methods has cost implications. 

While conducting observations and reviewing program docu- 
ments can be done with little additional funding, designing and 
administering a survey and analyzing its results may require 
the assistance of an outside evaluator. At minimum, it will 
require additional staff time for training. Similarly, using stan- 
dardized tests and assessments will likely necessitate external 
expertise to analyze results. An additional cost consideration 
is the use of management information systems (MIS) that can 
be used to record and store data collected from numerous 
methods. Like most other components of evaluation, it is 
important to start small and build data collection tools over 
time, as appropriate to the program's evolving evaluation 
needs. 

Suzanne Bouffard, Research Analyst 
Priscilla M. D. Little, Associate Director 
and Project Manager 
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NOTES 

1. Our database contains profiles of out-of-school time (OST) program 
evaluations, which are searchable on a wide range of criteria. It is avail- 
able in the OST section of the HFRP website at www.gse.harvard.edu/ 
hfrp/projects/afterschool/evaldatabase.html. 

2. For information on scales used to assess program quality, see 
Yohalem, N., Pittman, K., & Wilson-Ahlstrom, A. (2004). Getting inside 
the “black box" to measure program quality. The Evaluation Exchange, 

/ 0 ( I ), 6-7. Available atwww.gse.harvard.edu/hfrp/eval/issue2S/ 
spotlight.html. 



Documenting Progress and Demonstrating Results: 
Evaluating Local Out-of-School Time Programs, 

one of Harvard Family Research Project’s Issues and 
Opportunities in Out-of-School Time Evaluation briefs, 
provides practitioners of local out-of-school time 
programs with techniques, tools, and strategies for 
improving their programs and tracking their effective- 
ness over time, www.gse.harvard.edu/hfrp/content/ 
projects/afterschool/resources/issuebrief3.pdf 
(Acrobat file) 

Snapshot 3 in this series, Performance Measures 
in Out-of-School Time Evaluation, outlines 
the academic, youth development, and prevention 
performance measures currently being used by out- 
of-school time programs to assess their progress, and 
the corresponding data sources for these measures. 
www.gse.harvard.edu/hfrp/projects/afterschool/ 
resources/snapshot3.html 

Snapshot 6 on standardized assessments is 

scheduled to be released in fall 2004. To be notified 
when it is available sign up for our out-of-school 
time updates email at www.gse.harvard.edu/hfrp/ 
subscribe.html. 
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